Neural networks and neural memory

ABSTRACT

A neural pattern matcher is made up of an array of first sum and threshold SAT1 devices  18  each of which receives a number of inputs and a threshold value, and fires a 1 output if the number of inputs exceeds the threshold value. The outputs of the array of the SAT1 devices may be considered as a 2D image or generic template against which new data supplied into the registers  26  making up a data plane  24  are correlated at a correlation plane  20  of EX-NOR gates  22 . The outputs of the EX-NOR gates themselves may be summed and thresholded by a seconded sum and threshold device  28  to provide a neural output ‘1’ or ‘0’ indicating match or no match. The matcher may therefore behave as a neural auto-associative memory which continually adapts to the input data to recognize data of a particular specified class.

This invention relates to neural networks incorporating sum andthreshold devices and in particular, but not exclusively to suchnetworks capable of functioning as a neural pattern matcher. Theinvention also extends to sum and threshold devices for receivingweightless synaptic inputs and a weighted threshold value.

The apparatus and methods described herein may usefully incorporate,utilise, be used with or incorporated into any of the apparatus ormethods described in our co-pending U.K. Patent Application No.9726752.0 or our co-pending PCT Patent Applications Nos.PCT/GB98/______, PCT/GB98/______, PCT/GB98/______, (Our references03-7127, XA1154, XA1156 and XA1000), the entire contents of which areincorporated herein by reference.

Terminology

The term “Hamming value” is used to define the number of bits set in1-dimensional arrays such as a binary number, tuple, vector or 2 orhigher dimensional arrays, that is the number of 1's set. The Hammingvalue relationship of two binary numbers or arrays indicates which hasthe greater Hamming value or whether the Hamming values are the same.

The term “weighted binary” is used in the conventional sense to indicatethat successive bit positions are weighted, particularly . . . 16, 8, 4,2, 1 although other weighted representations are possible. “Weightlessbinary” is a set of binary digits 1 and 0, each representing just “1”and “0” respectively. There is no least significant bit (LSB) or mostsignificant bit (MSB). The set of bits may be ordered or without order.If all the 1's are grouped together e.g. [111000] then the code isreferred to as a thermometer code, thermocode or bar graph code, allcollectively referred to herein as “thermometer codes”. Equally, theterm thermometer code is used broadly to cover 1 or higher dimensionalarrays in which the set bits have been aggregated around a pre-set focalbit, which may be anywhere in the array.

A set of weightless bits is referred to herein as a “weightless tuple”or “weightless vector” and these terms are not intended to be restrictedto ordered sets.

In traditional neural networks, a real-valued synaptic value ismultiplied by a synaptic connection strength or weight value, and summedwith other similarly treated synapses before they are all summed andthresholded to form a neural output. The weight value is a real-valuedsynaptic connection strength and hence the common usage of the term“weighted neural network”. However, it is also possible to have binaryRAM-based neural networks that do not employ real-valued connectionweights but instead rely on the values of the binary bits being either 0or 1. Accordingly, there are two contexts of weightlessness: withoutsynaptic connection strength, and without binary code weighting. Thearrangements described herein employ weightless binary manipulationmechanisms and may be used to engineer weightless artificial neuralnetworks, otherwise referred to as weightless-weightless artificialneural networks.

In one context, this invention is concerned with the comparison of twoweightless vectors in terms of their Hamming values. This process isbroadly equivalent to the function of a binary neuron. If the neuronreceives a vector, A, of weightless synaptic values (e.g. [10110010]),and a vector, T, of weightless neural threshold values (e.g.[00101000]), the neuron may be required to fire because the Hammingvalue of A is greater than the Hamming value of T. In this example, thethreshold, T, can be thought of as a set of inhibitory synaptic valueswhich must be exceeded if the neuron is to be fired. The neuralnetworks, devices, and techniques disclosed herein may be used in flightcontrol systems, voting systems with redundancy, safety criticalsystems, telecommunications systems, decision making systems, andartificial intelligence systems, such as neural networks.

According to one aspect, this invention provides a neural networkcomprising:—

-   -   an array of bit memory means defining a neural memory for        storing binary bits representing a plurality of exemplars,    -   an array of sum and threshold devices each for receiving as        inputs respective bits from said bit memory means and for        providing a preset output if the sum of said inputs exceeds a        preset threshold, thereby to obtain a generic template        representing said exemplars, and    -   means for comparing or correlating a set of input data with said        generic template and providing an output representative of        extent of matching between said set of input data and said        generic template.

Preferably, said neural memory comprises means for storing a pluralityof 2-dimensional data arrays. The neural network preferably includesmeans for presenting said input data in parallel to a data plane forcorrelation with said generic template. The means for correlatingpreferably comprises an array of logic elements. In a preferredembodiment the network includes further sum and threshold means forcomparing the sum of said correlation results with a threshold and forproviding an output representative of a match, if said sum exceeds saidthreshold.

The sum and threshold devices may take many forms but the, or at leastone of the, sum and threshold devices preferably comprises a Hammingvalue comparator made up of a plurality of interconnected bitmanipulation cells, each bit manipulation cell being operable to effectat least one of a bit shift and a bit elimination operation.

In another aspect, this invention provides a device for providing anoutput representative of a sum and threshold function performed on aweightless input and a threshold value, which comprises means forconverting said weightless input into thermometer code (as hereindefined), and means for monitoring the bit at a bit positioncorresponding to said threshold.

In a further aspect, this invention provides a neural networkcomprising:—

-   -   an array of bit memory means defining a neural network for        storing binary bits representing a plurality of exemplars, and    -   an array of sum and threshold devices each for receiving as        inputs respective bits from said bit memory means and for        providing a preset output if the sum of said inputs exceeds a        preset threshold, thereby to obtain a generic template        representing said exemplars.

Whilst the invention has been described above, it extends to anyinventive combination of the features set out above or in the followingdescription.

The invention may be performed in various ways, and, by way of exampleonly, various embodiments thereof will now be described in detail,reference being made to the accompanying drawings which utilise theconventional symbols for logic gates and in which:—

FIG. 1 is a schematic diagram of a sum and threshold (SAT) device;

FIG. 2 is a diagram of a neural network for processing data inaccordance with this invention;

FIG. 3 is a diagram of a sum and threshold element with weightlesssynaptic inputs and a weighted binary threshold;

FIG. 4 is a diagram of a sum and threshold element of the type in FIG.3, employing a thermometer code converter; and

FIG. 5 is a diagram of a bit manipulator cell used in the thermometercode converter of FIG. 4.

The embodiments described herein make use of sum and threshold devices,which may take many forms. Examples of novel Hamming value comparatorswhich may serve as sum and threshold devices are described in ourcopending UK Patent Application No 9726752.0 and our copendingInternational Patent Application No. PCT/GB98/______ (Our referenceXA1154). Alternatively, the sum and threshold detectors may take theform of a binary thermometer code converter of the type described in ourco-pending International Patent Application No. PCT/GB98/______ (ourreference 03-7127) with a suitable output selector, as to be describedbelow. The Hamming Comparators and Binary Code Converters described inthese documents have the advantage that they can be implementedasynchronously, and thus be robust, fault tolerant and highly immune toRFI/EMI effects. However, of course, conventional sum and thresholddevices may also be used in carrying out this invention.

Referring now to FIG. 1, neural data and a neural threshold are suppliedto a Hamming value comparator 10 of one of the types discussed above,whether of one, two or of greater dimension. The input and thresholdtuples are weightless. The output of the comparator 10 indicates whetherthe neural data has exceeded the neural threshold. The output is thenviewed as the single output of the comparator or neuron which is takento have “fired” if the bit has set. In this sense, the Hamming valuecomparator 10 acts as a binary neuron.

FIG. 2 shows an embodiment of a neural pattern matcher 12 which uses anarray of sum and threshold devices, which may be those of the type justdescribed, or any other suitable SAT element. For ease of visualisation,this is illustrated as a 3-dimensional array, comprising a neural memoryblock 14 of dimensions w×d×m, where ‘m’ is the number of bits in thedigital input word, ‘w’ is the number of bits in the width of the inputpattern and ‘d’ is the number of exemplars in the neural memory. Thearrangement of neural memory is referred to elsewhere herein asneuroram.

The 3-dimensional array further includes a first sum and thresholdregion 16 (otherwise referred to as SAT columns) made up of (w×m) SATdevices 18 each marked SAT1 having d inputs (only one of these shown forclarity). Beneath the sum and threshold region 116 there is acorrelation plane 20 made up of (w×m) 2 input EX-NOR gates 22 each ofwhich receives an input from an associated SAT1 device 14 and an inputfrom an associated data plane 24 which, optionally, may be held inseparate bit memories 26. The outputs of the EX-NOR gates 22 in thecorrelation plane 20 are passed horizontally to a second sum andthreshold region (referred to as a SAT plane), which consist of a singleSAT2 device 28 with w×m inputs.

Respective thresholds T₁ of up to d bits are supplied to the SAT1devices 18, and a threshold T₂ of up to w×m bits is supplied to the SAT2device 28.

In use, incoming digital data is encoded using a unit Hamming distancecode such as thermometer code, Gray code, or is in the form of a halftone bit map etc. Data for training (or for recognition after learning)is presented in parallel to the input of the data plane 24, orsequentially using a planar shift register (not shown). Optionally, thecoded data may also be scrambled with a binary key string using EX-ORgates. The neural memory is programmed with ‘d’ exemplars each of w×mbits which can be prestored, learnt in a separate training routine, oradaptively altered during use.

The thresholds T₁ and T₂ determine the learning rate and quality factorof the neural pattern matcher/neural filter and are set according tofixed values, e.g. 66%. Alternatively the threshold may be adaptivelyset to determine the irritability (firing rate) of the system.Thresholds T₁ and T₂ may be thought of as a control over the degree ofconfidence attached to a match.

In considering operation of the device it is helpful to consider theaction of one of the SAT1 devices. If the threshold T₁ for thatparticular device is set at 66% of the number of exemplars (e.g. T₁ is 8if the neural memory is 12 bits deep), then the SAT1 device will providea set bit or “fire” if there are more than 8 bits set in the column ofneural memory above it. If T₁ is raised, then the SAT1 device will notfire until a greater proportion of the bits in the memory above it areset; in other words the degree of similarity of the bits in that columnmust be higher for the SAT1 to fire.

The outputs of all the (w×m) SAT1 devices may be considered as a generictemplate against which new data in the data plane is compared.

The threshold T₂ is an indicator of the degree of confidence insofar asthe greater T₂, the greater the number of correlations there have to bebetween the input data plane and the generic template plane for thepattern to be accepted.

The update of the neural memory can be “supervised” during a trainingphase in which “d” exemplars or patterns of the required class arepresented to the system and the neural memory then frozen.Alternatively, the update of neural memory can be unsupervised, so thatthe system continuously learns or adapts to incoming patterns after someinitial patterns have been fed in. Initially it may be wise to set theneural memory to random values when used in an unsupervised mode oftraining. To prevent drifting, a portion of the memory may be made “readonly”.

In this arrangement, the array may be regarded as a pattern matcher, ora category or class correlator. It is analogous in some respects to acortical column in physiological neural systems, and a group of arraysor neurorams is analogous to a cortical map. A group of neurorams can beordered linearly or in planes. Planar arrangements can be formed fromsub-patterns of neurorams, such as triangles, squares, hexagons, etc.The array provides a hardware embodiment of a neural auto-associativememory.

As a modification of the sum and threshold technique described above, aSum and Threshold element has been designed that accepts a weightlessbinary input and a weighted binary threshold. This Sum and Thresholdelement utilises a thermometer code converter, for example based on thethermometer code converter array described in our copending UK PatentApplication 9726752.0 or International Patent ApplicationPCT/GB98/______ (03-7127) and the appropriate output or outputs of thearray are monitored or processed in accordance with the value of thethreshold.

FIG. 3 shows a general arrangement of a sum and threshold element withweightless synaptic inputs and a weighted binary threshold. N weightlessbits are supplied to a thermometer code converter 30 to obtainthermometer code N_(t), which is passed to one or more selectors 32,each of which also receives a respective weighted threshold value T₁,T₂, etc. The selector 32 decodes the weighted threshold value and looksat the appropriate bit position in the thermometer code N_(t), and ifset (indicating that the thermometer code has a Hamming value greaterthan the specified threshold), the selector sets its output bit toindicate that N>T.

FIG. 4 is a circuit diagram of an example of a sum and threshold devicehaving eight synaptic inputs and a choice of four thresholds. The devicecomprises a thermometer code converter section 34, made up of 2-bitmanipulator cells 36 of the type shown in FIG. 5. The 2-bit manipulatorcells each comprise an OR gate 38 and an AND gate 40, interconnectedsuch that inputs a,b map to outputs Ya,Yb as follows:—

-   -   Ya=A OR B    -   Yb=A AND B

It should be noted that in the device of FIG. 4, there are eight inputs,thus requiring odd layers nominally of four 2-bit manipulator cells 36wide and even layers nominally of three 2-bit manipulator cells 38 wide,making up eight layers in all, although the fifth to eighth layers havebeen truncated in this case.

The thresholds in this example are I>6, I>5, I>4 and I>3, meaning thatonly the fourth to seventh outputs (from the bottom of the array asviewed) are required. Because of the truncation, the fifth to seventhlayers include AND is gates 42 at the lower truncation boundary, and theseventh layer includes an OR 44 gate.

The output of the thermometer code conversion section 34 passes to aweighted binary selector 46 which acts as a threshold decoder. Suchdevices are already known for use as binary multiplexers or logicalselectors. In this example, which allows selection of one of fourthreshold values 3, 4, 5, 6, the selector 40 comprises two weightedinputs 48, 50 which are each connected to the inputs of 3-input ANDgates 52, the other input of each AND gate being a respective outputfrom the thermometer code conversion section 34. Selected terminals ofthe lower three AND gates are inverted, and the outputs of the AND gatespass to an OR gate 54. Different permutations of 0's and 1's applied tothe weighted inputs select different bit positions at the output of thethermometer code conversion section 34.

The selector 46 has the following mapping:— Inputs Thermometer Code T₁T₀ Bit Position 0 0 4 0 1 5 1 0 6 1 1 7

Thus if the weighted input is (1,0) the device will fire only if theHamming value of the weightless input is greater than 5.

If two or more thresholds are to be determined, then further weightedbinary selectors could be connected to the output of the thermometercode converter, as shown in FIG. 3.

It will be appreciated also that the circuit could be simplified torespond to a given specific threshold; in this instance a binaryselector as such would not be required and instead the output of thethermometer code converter corresponding to T_(fixed)+1 where T_(fixed)is the fixed threshold, would be the output.

The devices described above provide robust hardware implementation whichis fault tolerant due to its weightless techniques.

In general, the implementation of the above arrangement is technologyindependent; electronic, ionic, magnetic or electromagnetic (e.g.optical) implementation are all suitable.

1-7. (Cancelled)
 8. A device for providing an output representative of asum and threshold function performed on a weightless input and athreshold value, which comprises means for converting said weightlessinput into thermometer code as herein defined, and means for monitoringthe bit at a bit position corresponding to said threshold.
 9. A devicefor providing an output representative of a sum and threshold functionperformed on a weightless input and a threshold value, which comprises aconverter for converting said weightless input into thermometer code asherein defined, and a monitor for monitoring the bit at a bit positioncorresponding to said threshold.